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Introduction 


This  work  was  done  as  part  of  a  larger  study  conducted  for  N814.  The 
purpose  of  the  larger  study  was  to  examine  the  link  between  mission 
performance  and  readiness  drivers  using  data  from  CVN-71's  combat 
operations  during  Operation  Allied  Force  (OAF).  In  this  part  of  the 
project,  we  looked  specifically  at  material  readiness  of  the  embarked 
airwing  (CVW-8). 

Our  original  intent  was  to  estimate  the  parameters  for  a  complete 
Markov  model  of  aircraft  material  condition.  The  transition  matrix 
shown  in  figure  1  gives  the  general  structure  of  such  a  model.  Each 
aircraft  was  to  be  considered  in  one  of  three  discrete  states:  airborne, 
not  airborne  but  mission  capable,  or  not  mission  capable.  Transition 
probabilities  between  the  states  were  to  work  as  shown  in  figure  1 .  For 
example,  pi  represents  the  probability  that  an  aircraft  that  is  not 
mission  capable  during  one  period  would  be  in  the  same  state  during 
the  next  period. 


Figure  1 .  Transition  matrix  representing  a  Markov 
model  of  aircraft  material  condition 
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We  were  unable  to  implement  a  complete  realization  of  this  model 
because  of  problems  that  included  missing  data  and  resource 
constraints.  However,  we  were  able  to  make  substantial  progress  on 
two  components  of  the  process  in  figure  1,  and  we  present  these 
results  below. 
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Data 


Data  sources  were  our  Maintenance  Action  Form  (MAF)  database  for 
information  on  sorties  and  transitions  between  states  for  individual  air¬ 
frames,  and  ISIS  data  that  allowed  us  to  link  pilots  to  particular  sorties. 
Information  on  aircraft  age  (for  F/A-18s)  came  separately  from 
NAVAIR.  For  reasons  that  we  do  not  understand,  NALCOLMIS  data  for 
this  battlegroup  are  not  available  for  April  of  1999;  we  are  therefore  lim¬ 
ited  to  May  and  early  June  as  the  only  periods  of  OAF  for  which  we  have 
data.  Summary  statistics  are  listed  in  table  1,  and  figure  2  shows  how  the 
sortie  durations  were  distributed  across  squadrons  and  over  time. 


Table  1 .  Summary  statistics 

Number  of  sorties 

814 

F— 1 4  sorties 

48.8% 

Training  sorties 

30.2% 

Support  sorties 

3.2% 

Percent  down  after  sorties 

25.3% 

Average  length  of  sorties 

2.35  hours 

Average  pilot  experience 

935  hours 

Average  a/c  age  (F/A-l  8s  only) 

7.25  years 

Average  down  spell  after  sortie 

1 7.9  hours 

Down-after-sortie  model 

The  failure-after-sortie  model  corresponds  to  probability  p4  in  figure  1. 
We  estimated  a  binary  dependent  variable  (probit)  model  where  the 
dependent  variable  was  whether  the  aircraft  went  to  a  "down"  status 
within  one  hour  of  completing  a  sortie.  Full  model  results  are  in  appen¬ 
dix  A,  but  our  principal  conclusions  are  as  follows: 

•  The  type  of  flight  mattered,  with  training  and  overhead  flights 
being  more  likely  to  result  in  a  subsequent  down  spell  than  opera¬ 
tional  flights.  However,  it  isn't  clear  that  this  relationship  is  directly 
causal,  at  least  for  the  training  flights.  Planes  that  were  due  to  go 
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Figure  2.  Distribution  of  sortie  duration  across  squadrons  and  over  time 
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down  later  anyway  might  well  be  those  that  were  designated  for 
training  activity.  Also,  most  of  the  overhead  flights  were 
functional  check  flights  that  occurred  immediately  after  a 
major  overhaul  or  repair.  Thus,  it  perhaps  should  come  as  no 
surprise  that  there  is  a  greater-than-usual  need  for  mainte¬ 
nance  work  after  the  check  flight  because  some  aspects  of  the 
overhaul  may  not  have  been  done  correcdy. 

•  There  were  marked  differences  between  squadrons,  and  again, 
it  isn't  clear  how  to  interpret  these  differences.  A  greater 
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tendency  to  take  a  plane  down  may  be  due  to  more  alert  crews, 
but  it  could  also  be  due  to  poor  earlier  work. 

•  Because  aging  platforms  are  an  increasing  source  of  concern  for 
the  Navy,  we  tried  to  identify  age  effects.  At  the  time  we  did  this 
work,  we  had  age  data  for  the  F/A-18s  in  CVW-8  only.  We  esti¬ 
mated  this  same  model  for  just  those  aircraft  and  included  age  as 
an  independent  variable.  For  F/A-18s,  the  model  produced  an 
estimate  that  an  additional  year  of  age  increased  the  propensity 
to  go  down  after  a  sortie  by  3.6  percent.  However,  this  result  was 
not  statistically  significant. 

•  We  included  sortie  length  and  pilot  characteristics  in  the  model, 
but  neither  of  these  had  a  statistically  significant  effect. 

•  We  can  get  a  rough  indication  of  whether  this  type  of  model  is  a 
good  fit  by  simply  counting  actual  and  predicted  outcomes. 
When  we  did  that  here,  we  found  that,  for  the  full  sample  of 
sorties,  there  were  ten  observations  (sorties)  where  the  estimated 
probability  of  a  plane  going  down  was  greater  than  50  percent.  In 
six  of  the  ten  sorties,  the  planes  did  in  fact  go  down  within  an 
hour  of  landing. 

Downtime  duration  model 

The  other  portion  of  a  Markov-type  model  that  we  examined  was  a 
duration  (hazard)  model  of  aircraft  downtime.  This  would  loosely  cor¬ 
respond  to  estimating  pi  in  figure  1.  We  estimated  it  using  the  data 
from  the  sortie  database — that  is,  we  used  only  those  down  spells  that 
were  attributed  to  sorties  in  the  model  of  the  previous  section.  There¬ 
fore,  this  model  doesn't  use  down  spells  for  aircraft  that  were  taken 
down  more  than  one  hour  after  they  returned  from  a  flight,  and, 
consequently,  it  doesn’t  fully  reflect  the  effects  of  routine  scheduled 
maintenance. 

We  present  complete  documentation  (LIMDEP  output)  in  appendix  B, 
but  this  is  a  summary  of  the  key  results: 

•  In  general,  sortie-specific  variables  had  little  effect  on  downtime. 
The  exception  was  if  the  sortie  was  for  training:  There  was  a  sta- 
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tistically  significant  increase  in  downtime  associated  with  sorties 
with  a  training  flight  purpose  code.  It  isn’t  clear  why  this  should 
be  so,  although  our  speculations  concerning  the  selection  of 
aircraft  for  training  purposes  may  be  appropriate  here  too. 

•  F-14s  stayed  down  longer  that  F/A-18s.  This  is  not  surprising 
because  the  F/A-18  is  well  known  for  being  relatively  easy  to 
work  on. 

•  The  age  effect  was  again  positive  and  not  statistically  significant. 

•  We  chose  the  Weibull  as  the  distribution  for  the  hazard  func¬ 
tion  because  of  its  generality.  (It  allows  for  either  an  increasing 
or  decreasing  hazard  function,  and  the  constant-hazard  special 
case  is  simply  the  exponential  distribution.)  From  the  actual 
model  estimation,  we  can  conclude  that  the  downtime  dura¬ 
tions  seem  to  follow  a  distribution  that  is  significantly  different 
from  the  exponential  and  has  a  decreasing  hazard.  This  is  con¬ 
sistent  with  previous  CNA  research  on  logistics  system 
performance.  (See  [2].) 

Summary  and  conclusions 

We  have  identified  some  of  the  variables  that  would  seem  to  be  rele¬ 
vant  to  the  determination  of  some  of  the  transition  probabilities  for  a 
Markov  model  of  aircraft  availability.  These  models  can  probably  be 
refined  even  further.  One  important  factor  that  was  not  allowed  for 
was  the  length  of  time  on  station;  this  would  likely  have  a  deleterious 
effect  on  both  people  and  machines.  Characteristics  of  the  individual 
maintainers  was  another  factor  that  we  could  not  incorporate  due  to 
data  limitations.  We  hope  to  be  able  to  match  maintainer  personnel 
data  to  MAFs  in  the  future. 

Note  that  there  is  a  considerable  similarity  between  the  framework  we 
are  considering  here  and  earlier  work  on  sortie-generation  models. 
(See  [3,  4].)  However,  in  those  models,  the  probability  distributions 
were  seen  as  essentially  fixed,  whereas  in  this  analysis,  we  are  trying  to 
allow  for  the  possibility  that  some  factors — “squawk  rates,”  for 
example — can  be  expected  to  vary  at  least  somewhat  in  response  to 
factors  that  we  can  measure. 
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Appendix 


Appendix  A.  Down-after-sortie  model  results 


We  did  all  our  statistical  modeling  with  the  LIMDEP  econometric 
software  package.  Text  output  from  the  down-after-sortie  model  fol¬ 
lows.  We  present  results  present  for  the  entire  CVW-8  fighter  and 
attack  population,  and  then  for  F/A-18s  only.  Most  of  the  variables 
are  self-explanatory,  but  two  of  them  merit  comment.  FLTHRSQR, 
which  is  the  square  of  flight  hours,  was  introduced  to  accommodate 
possible  nonlinearities  in  the  relationship.  SFTI  refers  to  the  rating 
system  for  pilots  discussed  in  [5,  pg.  56].  Hours  refers  to  the 
number  of  hours  the  pilot  had  flown  on  the  particular  T/M/S. 


I.  Combined  F-14s  and  F/A-18s 


Binomial  Probit  Model 
Maximum  Likelihood  Estimates 


|  Dependent  variable  UPORDOWN  j 

|  Weighting  variable  ONE  | 

|  Number  of  observations  814  | 

|  Iterations  completed  5  | 

|  Log  likelihood  function  -416.9280  | 

|  Restricted  log  likelihood  -460.4669  j 

|  Chi-squared  87.07775  | 

|  Degrees  of  freedom  9  | 

|  Significance  level  .0000000  | 

+ - + 


— 

-  + - +  - 

■ 

-  + - 

-+ - 

-  + - 1- 

Variable 

|  Coefficient  | 

Standard  Error 

| b/St . Er . 

.  I P [  I Z | >2] 

|  Mean  of  X| 

- 

-+ - + - 

Index  function  for  probability 

-  + - 

-+ - 

-  + - + 

Constant 

-1.478689920 

.48333903 

-3.059 

.  0022 

VF14 

.1788728074 

.13695153 

1.306 

.1915 

.26044226 

VF41 

- . 3  889  974744E-01 

.14187899 

-.274 

.7839 

.22727273 

VFA15 

-.9651991829 

.16151980 

-5.976 

.0000 

.28992629 

FLTHRS 

.4181767389 

.29599695 

1.413 

.1577 

2.3484029 

FLTHRSQR 

-  .  557  839  0117E-01 

. 44413665E-01 

-1.256 

.2091 

6.4334890 

TRAINING 

.3781438865 

.14247535 

2.654 

.0080 

.30221130 

SUPPORT 

.5890132056 

.30143126 

1.954 

.0507 

.  3 1941032E-0 1 

SFTI 

.  3169089817 E- 01 

.  57949122E-01 

.547 

.5845 

3.0294840 

HOURS 

.  1168902453E-03 

. 12453372E-03 

.939 

.3479 

934.63857 
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(Note:  E+nn  or  E-nn  means  multiply  by  10  to  +  or  -nn  power.) 


+ - + 

|  Partial  derivatives  of  E[y]  =  F[*]  with  | 


|  respect  to  the  vector  of  characteristics.  | 

|  They  are  computed  at  the  means  of  the  Xs .  | 

|  Observations  used  for  means  are  All  Obs .  | 

+ - + 


-  T  T  - 

-  T 

T  T 

Variable 

|  Coefficient  | 

Standard  Error 

| b/St . Er 

.  |P[ | Z | >z] 

|  Mean  of  x| 

Index  function  for  probability 

Constant 

- .  44218  517  84 

.14321719 

-3.088 

.  0020 

VF14 

.  5517  223277E-01 

.  43501586E-01 

1.268 

.2047 

.26044226 

VF41 

1153825076 E- 01 

.  41738818E-01 

-.276 

.7822 

.22727273 

VFA15 

-.2415348880 

.  31440892E-01 

-7.682 

.  0000 

.28992629 

FLTHRS 

.1250509341 

. 88455194E-01 

1.414 

.1574 

2.3484029 

FLTHRSQR 

1668153272 E- 01 

. 13274642E-01 

-1.257 

.2089 

6.4334890 

TRAINING 

.1189752113 

.  46683921E-01 

2.549 

.0108 

.30221130 

SUPPORT 

.2067590720 

.11701175 

1.767 

.0772 

.  31941032E-01 

SFTI 

.9476797851E-02 

.  17327290E-01 

.  547 

.5844 

3 . 0294840 

HOURS 

.  34954680 67E-04 

.  37259672E-04 

.938 

.3482 

934.63857 

(Note:  E+nn  or  E-nn  means 

multiply  by  10 

to  +  or  ■ 

-nn  power. 

) 

+ - + 

|  Fit  Measures  for  Binomial  Choice  Model  | 

I  Probit  model  for  variable  UPORDOWN  I 


+ - + 

|  Fit  Measures  for  Binomial  Choice  Model  | 

|  Probit  model  for  variable  UPORDOWN  | 

+ - + 

|  Proportions  P0=  .746929  Pl=  .253071  | 

|  N  =  814  N0=  608  Nl=  206  | 

|  LogL  =  -416.92804  LogLO  =  -460.4669  | 

4- - + 

|  Efron  |  McFadden  |  Ben./Lerman  | 

|  .09517  |  .09455  j  .65766  j 

|  Cramer  |  Veall/Zim.  |  Rsqrd_ML  | 

|  .09454  j  .18205  j  .10145  | 

+ - + 

j  Information  Akaike  I.C.  Schwartz  I.C.  | 

|  Criteria  1.04896  900.87569  | 

+ - + 


Frequencies  of  actual  &  predicted  outcomes 
Predicted  outcome  has  maximum  probability. 
Threshold  value  for  predicting  Y=1  =  .5000 
Predicted 
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Actual 

0 

1 

|  Total 

0 

604 

4 

|  608 

1 

200 

6 

j  206 

1 

I 

Total 

804 

10 

1  814 

-->  Probit;  lhs=Upordown;  rhs=one/ acage, vfal5 , Flthrs , f lthrsqr ,  training, 
supp . .  . 

Normal  exit  from  iterations.  Exit  status=0. 

II.  F/A-18s  only. 


+ - + 

|  Binomial  Probit  Model  | 

|  Maximum  Likelihood  Estimates  | 

|  Dependent  variable  UPORDOWN  | 

|  Weighting  variable  ONE  | 

|  Number  of  observations  417  | 

|  Iterations  completed  5  | 

|  Log  likelihood  function  -168.7557  | 

|  Restricted  log  likelihood  -193.4127  | 

|  Chi-squared  49.31406  | 

|  Degrees  of  freedom  8  | 

|  Significance  level  .0000000  | 

+ - + 


Variable 

|  Coefficient 

Standard  Error 

| b/St . Er 

.  |P[  |Z|>z] 

|  Mean  of  X| 

Constant 

Index  function  for  probability 

-2.446023800  1.5340101 

-1.595 

.1108 

ACAGE 

.1593963571 

.16982052 

.939 

.3479 

7.2481439 

VFA15 

-.9711541856 

.17341141 

-5.600 

.0000 

.56594724 

FLTHRS 

.4450233708 

.61628718 

.722 

.4702 

2.1491607 

FLTHRSQR 

-.7614385694E-01 

.  12148838 

-.627 

.5308 

4.9658993 

TRAINING 

.1825954579 

.22798726 

.801 

.4232 

.33093525 

SUPPORT 

.6838744847 

.45083337 

1.517 

.1293 

.  33 573 141E-01 

SFTI 

. 9080547778 E- 01 

.  72121584E-01 

1.259 

.2080 

2.8848921 

HOURS 

-.1789763417E-03 

. 18051210E-03 

-.991 

.3214 

847.77410 

(Note:  E+nn  or  E-nn  means 

multiply  by  10 

to  +  or  - 

-nn  power . ' 

) 
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+ - + 

|  Partial  derivatives  of  E[y]  =  F [ * ]  with  | 


|  respect  to  the  vector  of  characteristics.  | 

|  They  are  computed  at  the  means  of  the  Xs .  j 
|  Observations  used  for  means  are  All  Obs .  | 
+ - + 


"  T 

- 

- 1 - r 

Variable 

|  Coefficient  | 

Standard  Error 

|b/St . Er 

tsi 

V 

N 

Mean  of  X| 

Index  function  for  probability 

Constant 

-.5514477497 

.34376591 

-1.604 

.1087 

ACAGE 

. 3593536678 E- 01 

.  38330485E-01 

.938 

.3485 

7.2481439 

VFA15 

-.2338379454 

.  41770843E-01 

-5.598 

.0000 

.56594724 

FLTHRS 

.1003290060 

.13831590 

.725 

.4682 

2.1491607 

FLTHRSQR 

-. 1716637367 E- 01 

.  27281959E-01 

-.629 

.5292 

4.9658993 

TRAINING 

. 4253006984 E- 01 

.  54656587E-01 

.778 

.4365 

.33093525 

SUPPORT 

.2042692970 

.16251556 

1.257 

.2088 

* 33573141E-01 

SFTI 

.  2047178624E-01 

.  16265249E-01 

1.259 

.2082 

2 . 8848921 

HOURS 

-. 4034960776 E- 04 

.  40713239E-04 

-.991 

.3217 

847.77410 

(Note:  E+nn  or  E-nn  means 

multiply  by  10 

to  +  or  - 

-nn  power , 

) 

+ - + 

|  Fit  Measures  for  Binomial  Choice  Model  | 

|  Probit  model  for  variable  UPORDOWN  j 

+ - + 

|  Proportions  PQ=  .824940  Pl=  .175060  | 

j  N  =  417  N0=  344  Nl=  73  | 

|  LogL  =  -168.75566  LogLO  =  -193.4127  | 

+ - + 

|  Efron  |  McFadden  |  Ben./Lerman  | 

j  .11871  j  .12748  |  .74516  j 

|  Cramer  |  Veall/Zim.  |  Rsqrd_ML  | 

|  .11779  j  .21976  j  .11153  | 

+ - + 

|  Information  Akaike  I.C.  Schwartz  I.C.  | 

|  Criteria  .85255  391.80909  | 

+ - v 


Frequencies  of  actual  &  predicted  outcomes 
Predicted  outcome  has  maximum  probability. 
Threshold  value  for  predicting  Y=1  =  .5000 
Predicted 

-  -  +  - 
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Actual 

0 

1 

|  Total 

0 

1 

343 

72 

1 

1 

|  344 

j  73 

Total 

415 

2 

1  417 
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Appendix  B.  Downtime  hazard  model  results 


Here  we  give  the  text  output  for  the  downtime  hazard  model.  As  in 
appendix  A,  we  present  combined  F-14  and  F/A-18  results  first, 
followed  by  separate  results  for  F/A-18s  only.  Note  that  it  is  necessary 
to  take  the  natural  logarithm  of  downtime  for  use  in  this  routine.  The 
“sigma”  in  this  output  is  the  parameter  that  determines  the  slope  of 
the  hazard;  the  fact  that  it  is  significantly  different  from  one  in  the 
first  regression  establishes  that  the  hazard  function  in  that  case  is  not 
exponential.  (It  is  borderline  significant  in  the  second  regression.) 

I.  Combined  results  for  F-14s  and  F/A-18s 


+ - + 

|  Loglinear  survival  model:  WEIBULL  | 

|  Maximum  Likelihood  Estimates  | 

|  Dependent  variable  LNDWNTIM  | 

|  Weighting  variable  ONE  | 

|  Number  of  observations  208  | 

|  Iterations  completed  14  | 

|  Log  likelihood  function  -417.9449  | 

+ - + 


+ - + - + - + - + - + - + 

(Variable  |  Coefficient  |  Standard  Error  | b/St . Er . | P [ | Z | >z]  |  Mean  of  X| 

+ - + - + - + - + - + - + 

RHS  of  hazard  model 


Constant 

2.496620695 

1.0964933 

2.277 

.0228 

VF14 

-1.659409229 

.34949381 

-4.748 

.0000 

.42788462 

VF41 

-.6713862412 

.33514312 

-2.003 

.0451 

.28846154 

VFA15 

-1.110724834 

.87002803 

-1.277 

.2017 

. 432  692  31E-01 

FLTHRS 

.7252487158 

.70132916 

1.034 

.3011 

2.4639423 

FLTHRSQR 

-.1399716072 

.10076648 

-1.389 

.1648 

7.3610096 

TRAINING 

-.8916738957 

.31887733 

-2.796 

.0052 

.35096154 

SUPPORT 

-.2672918828 

.54106707 

-.494 

.6213 

.  52884615E-01 
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Ancillary  parameters  for  survival 
Sigma  1.691159392  . 97149294E-01  17.408  .0000 

(Note:  E+nn  or  E-nn  means  multiply  by  10  to  +  or  -nn  power.) 


4- - + 

|  Parameters  of  underlying  density  at  data  means:  | 

I  Parameter  Estimate  Std.  Error  Confidence  Interval  ! 


.1822 
.  6579 
5.0848 


-->  Reject?  age  <  0  $ 

-->  Survival;  lhs=LnDwnTim? 

rhs=one/ age, vfal5 , Flthrs , f lthrsqr ,  training,  support; 
model=Weibull  $ 

Normal  exit  from  iterations.  Exit  status^O . 


Lambda 

P 

Median 

Percentiles 

Survival 

Time 


.13884  .02211  .0955  to 

.59131  .03397  .5247  to 

3.87515  .61717  2.6655  to 

of  survival  distribution: 

.25  .50  .75  .95 

12.51  3.88  .88  .05 


II.  Results  for  F/A-18s  only 


Loglinear  survival  model:  WE I BULL 
Maximum  Likelihood  Estimates 
Dependent  variable  LNDWNTIM 

Weighting  variable  ONE 

Number  of  observations  59 

Iterations  completed  14 

Log  likelihood  function  -108.6682 
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Variable 

|  Coefficient  |  Standard  Error  | ' 

b/St.Er. 

.  1  P [ 1 Z I >z] 

|  Mean  of  X| 

Constant 

RHS  of  hazard  model 

.7007576221 

4.8927149 

.  143 

.8861 

AGE 

.5270809643 

.39845108 

1.323 

.1859 

7.3998805 

VFA15 

-.9520432114 

.65406279 

-1.456 

.  1455 

.15254237 

FLTHRS 

-.2352836102 

3.1695339 

-.074 

.9408 

2 . 0474576 

FLTHRSQR 

-.5385881816E-01 

.76223681 

-.071 

.9437 

4.5077965 

TRAINING 

-1.545007594 

.78947811 

-1.957 

.0503 

.40677966 

SUPPORT 

-2.793202177 

1.5556986 

-1.795 

.0726 

.  84745763E-01 

Sigma 

Ancillary  parameters 

1.302560425 

for  survival 

.16626092 

7.834 

.0000 

(Note:  E+nn  or  E-nn  means  multiply  by  10  to  +  or  -nn  power.) 


+ - + 

|  Parameters  of  underlying  density  at  data  means:  | 

I  Parameter  Estimate  Std.  Error  Confidence  Interval  I 


Lambda  .05692  .01085  .0357  to  .0782  | 

P  .76772  .09799  .5757  to  .9598  | 

Median  10.89991  2.07784  6.8273  to  14.9725  | 

Percentiles  of  survival  distribution:  ; 

Survival  .25  .50  .75  .95  | 

Time  26.89  10.90  3.47  .37  | 

+ - + 
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